Papers with Human evaluation
No Reader Left Behind: Multi-Agent Summaries Everyone Can Understand (2026.acl-long)
Copied to clipboard
| Challenge: | Existing summarization systems struggle to address diverse linguistic and cognitive barriers among general readers. |
| Approach: | They propose a multi-agent framework that integrates template-based planning with an iterative feedback loop guided by simulated readers and domain expert revision to address comprehension barriers such as unknown terms, missing contexts, and confusing sentences. |
| Outcome: | The proposed framework improves readability and factuality across multiple datasets and human evaluations show that it is more accessible to a wide range of readers. |
KreolMorisienMT: A Dataset for Mauritian Creole Machine Translation (2022.findings-aacl)
Copied to clipboard
| Challenge: | Mauritian Creole is a French-based creole and a lingua franca of the Republic of Mauritius. |
| Approach: | They describe a dataset for benchmarking machine translation quality of Mauritian Creole. |
| Outcome: | The proposed dataset compares KreolMorisienMT with existing models and human evaluation reveals the systems’ high translation quality. |
PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation (2023.eacl-main)
Copied to clipboard
| Challenge: | Curriculum Data Augmentation (CDA) presents synthetic data with increasing difficulties to neural models. |
| Approach: | They propose a curriculum-aware paraphrase generation module with bottom-k sampling and cyclic learning strategy that passes through the curriculums multiple times. |
| Outcome: | The proposed framework surpasses competitive baselines on few-shot text classification and dialogue generation. |
Catching Attention with Automatic Pull Quote Selection (2020.coling-main)
Copied to clipboard
| Challenge: | Using a novel task, we advocate automatic pull quote selection to engage readers with thought-provoking articles . pull quotes increase enjoyment and readability, shape reader perceptions, and facilitate learning. |
| Approach: | They propose a task that automatically selects pull quotes from articles with more salient presentation. |
| Outcome: | The proposed task differs from similar tasks such as summarization and clickbait identification by several aspects. |
Leveraging Implicit Feedback from Deployment Data in Dialogue (2024.eacl-short)
Copied to clipboard
| Challenge: | Xu et al., 2023) and Bai ed., 2019) use crowdworkers to collect signals from natural dialogue episodes. |
| Approach: | They use the publicly released BlenderBot deployment data to extract signals from conversations to implicitly measure the quality of a machine-generated utterance. |
| Outcome: | The proposed model improves over baseline models, but some proxy signals can lead to undesirable generations. |
RankME: Reliable Human Ratings for Natural Language Generation (N18-2)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluation for natural language generation often suffers from inconsistent user ratings. |
| Approach: | They propose a rank-based magnitude estimation method which combines continuous scales and relative assessments to improve the reliability of human ratings. |
| Outcome: | The proposed method significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. |
MULSUM: A Multimodal Summarization System with Vis-Aligner and Diversity-Aware Image Selection (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing systems that condense text and images into concise, faithful digests are inefficient and require large fusion transformers. |
| Approach: | They propose a framework that uses image embeddings to generate a visually informed text summary and a Diversity-Aware Image Selector to maximize images-relevance to the summary. |
| Outcome: | The proposed framework outperforms baselines on automatic metrics such as ROUGE and human evaluation shows that selected images act as explanatory evidence rather than ornamental add-ons. |
Fluent Response Generation for Conversational Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Question answering (QA) is an important aspect of open-domain conversational agents, garnering specific research focus in the conversational QA subtask. |
| Approach: | They propose a method for situating QA responses within a SEQ2SEQ NLG approach to generate fluent grammatical answer responses while maintaining correctness. |
| Outcome: | The proposed model outperforms baseline CoQA and QuAC models in generating conversational responses. |
Chandomitra: Towards Generating Structured Sanskrit Poetry from Natural Language Inputs (2026.eacl-long)
Copied to clipboard
Manoj Balaji Jagadeeshan, Samarth Bhatia, Pretam Ray, Harshul Raj Surana, Akhil Rajeev P, Priya Mishra, Annarao Kulkarni, Ganesh Ramakrishnan, Prathosh Ap, Pawan Goyal
| Challenge: | Large language models are capable of creative generation tasks but prominently for high-resource languages. |
| Approach: | They propose to use large language models for structured poetry generation in Sanskrit . their constrained decoding method achieves 99.86% syntactic accuracy . |
| Outcome: | The proposed model outperforms the existing model in generating metrically valid Sanskrit poetry. |
Samvaadhana: A Telugu Dialogue System in Hospital Domain (D19-61)
Copied to clipboard
| Challenge: | a dialogue system for Hospital domain in Telugu is a resource-poor Dravidian language . the system handles various hospital and doctor related queries . |
| Approach: | They propose to model a dialogue system for Hospital domain in Telugu which is a resource-poor Dravidian language. |
| Outcome: | The proposed system achieves a high overall rating and a significantly accurate context-capturing method. |
Controlled Language Generation for Language Learning Items (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have resulted in success in generating fluent English text. |
| Approach: | They propose to employ natural language generation to rapidly generate English language items . they experiment with deep pretrained models and develop methods for controlling items for factors relevant in language learning . |
| Outcome: | The proposed framework shows high grammatically scores for all models and higher complexity over baseline models. |
Identifying High Consideration E-Commerce Search Queries (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Identifying high consideration queries is essential for e-commerce sites to better serve user needs . ecommerce sites can create or serve customized content for specific queries . |
| Approach: | They propose an engagement-based Query Ranking approach to identify potential engagement levels with query-related shopping knowledge content during product search. |
| Outcome: | The proposed method outperforms human-selected queries in terms of customer impact . human evaluation shows a precision of 96% for HC queries identified by the model . |
Abstractive Document Summarization with Summary-length Prediction (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing abstractive summarization models do not consider summarizing-specific information such as the target summary length. |
| Approach: | They propose a method for enabling a model to understand summarization-specific information by predicting the summary length in the encoder and generating a summary of the predicted length in fine-tuning. |
| Outcome: | The proposed method improves ROUGE scores on the WikiHow, NYT, and CNN/DM datasets. |
Where to Attack: A Dynamic Locator Model for Backdoor Attack in Text Classifications (2022.coling-1)
Copied to clipboard
| Challenge: | BackDoor Attack (BDA) study aims to train a poisoned model with clean data and some trigger-embedded instances to perform normally on normal inputs. |
| Approach: | They propose to train a poisoned model with clean and poisonest inputs . they propose to use triggers to predict those poisonets as target labels . |
| Outcome: | The proposed model can predict P2P dynamically without human intervention. |
Improving Reinforcement Learning Based Image Captioning with Natural Language Prior (D18-1)
Copied to clipboard
| Challenge: | Recent research shows that Reinforcement Learning (RL) approaches suffer from the exposure bias problem. |
| Approach: | They propose a Reinforcement Learning (RL) based training framework that constrains the action space using an n-gram language prior. |
| Outcome: | The proposed model is more human readable and graceful. |
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021.tacl-1)
Copied to clipboard
| Challenge: | a large study of machine translation systems shows poor evaluation procedures can lead to erroneous conclusions. |
| Approach: | They propose an evaluation methodology grounded in explicit error analysis based on the Multidimensional Quality Metrics framework. |
| Outcome: | The proposed evaluation methodology outperforms crowd workers in two languages . it shows that human-based metrics outperformed crowd workers . |
Towards Document-Level Paraphrase Generation with Sentence Rewriting and Reordering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level paraphrase generation is an important task in natural language processing. |
| Approach: | They propose a coherence relationship-guided paraphrase generation model that leverages graph GRU to encode the coherency relationship graph and get the cohesion-aware representation for each sentence. |
| Outcome: | The proposed model outperforms baseline models on BERTScore and diversity scores. |
The Mechanical Bard: An Interpretable Machine Learning Approach to Shakespearean Sonnet Generation (2023.acl-short)
Copied to clipboard
| Challenge: | Rather than train a model to obey these constraints implicitly, we opt to enforce them explicitly using a simple but novel approach to generation. |
| Approach: | They propose to automate the generation of sonnets within preset poetic constraints using a constrained decoding approach that uses a relatively modest neural backbone. |
| Outcome: | The proposed method produces sonnets that adhere to the genre’s defined constraints and contain lyrical language and literary devices. |
Learning to Write with Cooperative Discriminators (P18-1)
Copied to clipboard
| Challenge: | Despite their local fluency, long-form text generated from RNNs is often generic, repetitive, and even self-contradictory. |
| Approach: | They propose a unified learning framework that can guide a base RNN generator towards more globally coherent generations by combining discriminators with a composite decoding objective. |
| Outcome: | The proposed framework can guide a base RNN generator towards more globally coherent generations by combining discriminators with the base RRN generator through a composite decoding objective. |
ExCAR: Event Graph Knowledge Enhanced Explainable Causal Reasoning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work infers the causation between events based on knowledge from annotated causal event pairs, but additional evidence information is unexploited. |
| Approach: | They propose an Event graph knowledge enhanced explainable CAusal Reasoning framework that acquires additional evidence information from a large-scale causal event graph as logical rules for causal reasoning. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in human evaluation and in animal models. |
Complex Claim Verification with Evidence Retrieved in the Wild (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work makes simplifying assumptions in retrieval that depart from real-world use cases: no access to evidence, access to curated evidence, or access to published evidence after a claim was made. |
| Approach: | They propose a pipeline to check claims using raw evidence from the web . they restrict their retriever to only search documents available prior to the claim's making . |
| Outcome: | The proposed method is based on a political claim dataset and shows that the evidence summary produced by the system is reliable and relevant to answering key questions. |
Don’t Generate, Classify! Low-Latency Prompt Optimization with Structured Complementary Prompt (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated strong performance across diverse tasks, but their performance varies significantly across different prompts. |
| Approach: | They propose a framework that reframes prompt engineering as a classification problem. |
| Outcome: | The proposed framework improves answer quality by up to 26.5% in win rate compared to prior methods while reducing latency by upto 1,956 times. |
Learning with Contrastive Examples for Data-to-Text Generation (2020.coling-main)
Copied to clipboard
Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao
| Challenge: | Existing models for data-to-text generation generate fluent but sometimes incorrect sentences . Existing studies show that using contrastive examples improves the ability of generating sentences with better lexical choice without degrading the fluency. |
| Approach: | They propose to use models trained on incorrect sentences and learning methods that exploit contrastive examples to reduce such errors. |
| Outcome: | The proposed models generate fluent sentences but often have problematic ones in terms of correctness. |
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models (2020.emnlp-main)
Copied to clipboard
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
| Challenge: | Existing pre-trained large language models have shown unparalleled generative capabilities, but they are not controllable. |
| Approach: | They propose a framework that uses large-scale language models and adds control to text generation by incorporating an external knowledge base. |
| Outcome: | The proposed model generates more fluent, consistent, and coherent stories with less repetition and higher diversity compared to previous work on the ROC story dataset. |
Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | reproducibility of human evaluations is rarely queried in NLP . authors estimate that just 5% of humanevaluations are repeatable . |
| Approach: | They propose to make human evaluations more repeatable and more reproducible . they estimate that just 5% of human evaluation experiments are repeatable . |
| Outcome: | The results show that human evaluations are rarely queried or formally tested in NLP . the authors estimate that just 5% of human evaluation experiments are repeatable . |
Modeling Intensification for Sign Language Generation: A Computational Approach (2022.findings-acl)
Copied to clipboard
| Challenge: | End-to-end sign language generation models do not accurately represent prosody in sign language. |
| Approach: | They propose to model intensification in a data-driven manner to improve prosody in generated sign languages by modeling temporal and spatial variations. |
| Outcome: | The proposed models improve the prosody of generated sign languages by using data-driven models. |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open-domain dialogue systems conduct one-session conversations, but multi-session MSCs are under-investigated. |
| Approach: | They propose a History-Aware Hierarchical Transformer for multi-session open-domain dialogue . they propose to encode history conversations into a history memory and leverage historical information to generate well-informed responses. |
| Outcome: | The proposed model outperforms baseline models on a large-scale MSC dataset. |
Argument Generation with Retrieval, Planning, and Realization (P19-1)
Copied to clipboard
| Challenge: | a novel argument generation framework is used to generate counter-arguments . CANDELA uses a text planning decoder to retrieve arguments of different perspectives . |
| Approach: | They propose a powerful retrieval system and a novel two-step argument generation framework . they use a retrieval-based retrieval platform indexed with 12 million articles from Wikipedia . |
| Outcome: | The proposed framework yields higher BLEU, ROUGE, and METEOR scores than state-of-the-art models. |
On the Interaction of Belief Bias and Explanations (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate explainability fail to account for belief biases affecting human performance . previous studies have shown that neural models can make confident predictions relying on artifacts . |
| Approach: | They propose to account for belief bias in explainability by using models of varying quality and adversarial examples. |
| Outcome: | The proposed methods show that results change when using models of varying quality and adversarial examples. |
Let’s Simplify Step by Step: Guiding LLM Towards Multilingual Unsupervised Proficiency-Controlled Sentence Simplification (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models demonstrate limited capability in proficiency-controlled sentence simplification when simplifying across large readability levels. |
| Approach: | They propose a framework that decomposes complex simplifications into manageable steps through dynamic path planning, semantic-aware exemplar selection, and chain-of-thought generation with conversation history for coherent reasoning. |
| Outcome: | The proposed framework reduces computational steps while improving simplification effectiveness on five languages across two benchmarks. |
Non-Sequential Graph Script Induction via Multimedia Grounding (2023.acl-long)
Copied to clipboard
| Challenge: | Existing scripts for everyday tasks are presented in a linear manner, which does not reflect the flexibility displayed by people executing tasks in real life. |
| Approach: | They propose to use loosely aligned videos to train a non-sequential graph script induction task by using a multimodal framework to ground procedural videos to WikiHow textual steps. |
| Outcome: | The proposed model outperforms the WikiHow linear baseline by 48.76% . it can predict future steps given a partial step sequence and generate explicit graph scripts . |
Better Rewards Yield Better Summaries: Learning to Summarise Without References (D19-1)
Copied to clipboard
| Challenge: | Reinforcement Learning (RL)-based document summarisation systems produce state-of-the-art performance in terms of ROUGE scores, but high summaries receive low human judgement. |
| Approach: | They propose to learn a reward function from human ratings on 2,500 summaries to generate human-appealing summary. |
| Outcome: | The proposed reward function can generate human-appealing summaries without reference summary input. |
Distill and Align Decomposition for Enhanced Claim Verification (2026.findings-eacl)
Copied to clipboard
Jabez Magomere, Elena Kochkina, Samuel Mensah, Simerjot Kaur, Fernando Acero, Arturo Oncevay, Charese Smiley, Xiaomo Liu, Manuela Veloso
| Challenge: | Existing methods for complex claim verification struggle to align decomposition quality with verification performance. |
| Approach: | They propose a reinforcement learning approach that optimizes decomposition quality and verifier alignment using Group Relative Policy Optimization. |
| Outcome: | The proposed method outperforms prompt-based approaches and existing methods in six evaluation settings. |
CollabKG: A Learnable Human-Machine-Cooperative Information Extraction Toolkit for (Event) Knowledge Graph Construction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing IE tools lack multi-task support and automatic updates for KG and EKG construction. |
| Approach: | They propose a human-machine-cooperative IE toolkit for KG and EKG construction that unifies different IE subtasks and integrates LLMs as the assistant machine. |
| Outcome: | The proposed tool improves annotation quality, efficiency, and stability simultaneously. |
Controllable Dialogue Simulation with In-context Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate annotated dialogues require crowdsourcing, which is expensive and time-consuming. |
| Approach: | They propose a dialogue simulation method based on large language model in-context learning that generates new dialogues and annotations in a controllable way. |
| Outcome: | The proposed method can expand a small set of dialogue data with minimum or zero human involvement and parameter update. |
Harnessing Black-Box Control to Boost Commonsense in LM’s Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen remarkable progress in massively Pre-Trained Language Models such as GPT-3 . however, their generated outputs lack commonsense at times . |
| Approach: | They propose a framework that steers a frozen Pre-Trained Language Model towards more commonsense generation by training an auxiliary model. |
| Outcome: | The proposed framework produces plausible outputs that incorporate concepts in a meaningful way. |
Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents. |
| Approach: | They propose a nested tree-based extractive summarization model on RoBERTa that uses syntactic and discourse trees to represent sentences in a given document. |
| Outcome: | The proposed model outperforms baseline models on the CNN/DailyMail dataset and achieves significantly better scores than the baseline models in terms of coherence and comparable scores to the state-of-the-art models. |
SgSum:Transforming Multi-document Summarization into Sub-graph Selection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing extractive multi-document summarization methods score each sentence individually and extract salient sentences one by one. |
| Approach: | They propose a novel framework for extractive multi-document summarization that selects a sub-graph as the summary instead of selecting salient sentences. |
| Outcome: | The proposed framework improves on existing methods on multi-document datasets and human evaluations show it produces more coherent and informative summaries. |
MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding (2021.naacl-main)
Copied to clipboard
| Challenge: | a new method for generating metaphors is proposed to generate literal sentences . human evaluations show that our best model generates metaphors better than three well-crafted baselines 66% of the time on average. |
| Approach: | They propose a method to automatically construct a parallel corpus by transforming literal sentences to metaphorical ones using commonsense inference and masked language modeling. |
| Outcome: | The proposed method generates metaphors better than baselines 66% of the time on average. |
Neural Stylistic Response Generation with Disentangled Latent Variables (2021.acl-long)
Copied to clipboard
| Challenge: | Existing parallel datasets for creating stylistic responses are not stylistically consistent. |
| Approach: | They propose to disentangle the content and style in latent space by diluting sentence-level information in style representations. |
| Outcome: | The proposed approach achieves a higher BERT-based style intensity score and comparable BLEU scores, compared with baselines. |
Progressive Generation of Long Text with Pretrained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for "long" text generation are limited to outputs of 50-200 tokens . however, our proposed ProGen generates coherent long passages of text in a progressive manner . |
| Approach: | They propose a method for generating coherent long passages of text in a progressive manner . they first produce domain-specific content keywords and then refine them into complete passages . human evaluation validates that their proposed generation is more coherent . |
| Outcome: | The proposed method produces domain-specific content keywords and refines them into complete passages in multiple stages. |
Beyond Facts: Evaluating Intent Hallucination in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) produce unsatisfactory results when faced with complex queries containing multiple conditions. |
| Approach: | They propose a benchmark for intent hallucination that covers 20,068 problems and an automatic LLM generation evaluation metric for detecting intent hallucinosis. |
| Outcome: | The proposed benchmark covers query-only and retrieval-augmented generation (RAG) setups with varying topics and difficulty. |
Generating Dialogue Responses from a Semantic Latent Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for dialogue generation are unable to integrate information from multiple semantically similar valid responses of a given prompt. |
| Approach: | They propose to learn the pair relationship between the prompts and responses as a regression task instead of the end-to-end classification on vocabulary. |
| Outcome: | The proposed model learns the pair relationship between the prompts and responses on a latent space instead of the end-to-end classification on vocabulary. |
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms that generate captions for scientific figures are costly and dependent on author-written captions. |
| Approach: | They constructed a human evaluation dataset that contains human judgments for 3,600 scientific figure captions for 600 arXiv figures. |
| Outcome: | The proposed model outperforms all other models and outperformed undergraduates in achieving a Kendall correlation score of 0.401 with Ph.D. students’ rankings. |
ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024.acl-long)
Copied to clipboard
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song
| Challenge: | Existing evaluation methods for large language models are labor-intensive and lack efficiency. |
| Approach: | They propose a framework dedicated to assessing long-text generation that includes in-depth human-curated meta-questions spanning various domains . they use a set of proxy-quests with pre-annotated answers to assess the content's quality by incorporating the generated texts as contextual background. |
| Outcome: | The proposed framework assesses the quality of long-text content by matching it with references through human evaluation or automated metrics. |
Does Meta-learning Help mBERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages? (2022.coling-1)
Copied to clipboard
Aniruddha Roy, Rupak Kumar Thakur, Isha Sharma, Ashim Gupta, Amrith Krishna, Sudeshna Sarkar, Pawan Goyal
| Challenge: | Existing approaches to few-shot Question Generation (QG) are limited and require manual annotation. |
| Approach: | They propose to use multilingual BERT to perform few-shot question generation with cross-lingual transfer. |
| Outcome: | The proposed model improves in few-shot QG and human evaluation confirms it. |
PersonaForge: Psychology-Grounded Dual-Process Architecture for Personality-Consistent Role-Playing Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to role-playing with Large Language Models lack consistency across long conversations. |
| Approach: | They propose a three-layer personality architecture grounded in psychological theory and a dual-process generation mechanism inspired by cognitive science to solve this problem. |
| Outcome: | The proposed framework reduces drift over 50-turn conversations by reducing personality consistency . human evaluation confirms more authentic and psychologically coherent character behaviors. |
FRAME: Feedback-Refined Agent Methodology for Enhancing Medical Research Insights (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to automate scientific research are limited by human cognitive constraints and timeintensive workflows. |
| Approach: | They propose a framework that enhances medical paper generation through iterative refinement and structured feedback. |
| Outcome: | The proposed framework achieves significant improvements over conventional methods across multiple models and evaluation dimensions. |
SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing literature review models have addressed literature review generation, but lack of large-scale datasets has been a stumbling block. |
| Approach: | They propose to use a large-scale dataset to evaluate automatic literature review generation models. |
| Outcome: | The proposed model can generate summaries comparable to human-written reviews while lacking detailed information. |
Hierarchical Trivia Fact Extraction from Wikipedia Articles (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for extracting trivia facts for Wikipedia categories are not efficient . a trivia fact is an interesting fact that is unusual, unexpected, or unique . |
| Approach: | They propose an unsupervised algorithm that automatically mines trivia facts for a given entity . they propose to target at a single Wikipedia article and leverage its hierarchical structure . |
| Outcome: | The proposed algorithm outperforms existing methods and is 100 times faster than existing methods. |
Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detect false claims ignore the characteristics of FC-articles . claims are often quoted to describe checked events, providing lexical information . sentence templates to introduce or debunk claims are common across articles, providing pattern information. |
| Approach: | They propose a model to rerank FC-articles using key sentences and pattern information. |
| Outcome: | The proposed model outperforms existing methods on two real-world datasets showing that key sentences can be used to predict if an article fact-checks the given claim. |
A Human-machine Collaborative Framework for Evaluating Malevolence in Dialogues (2021.acl-long)
Copied to clipboard
| Challenge: | Automatic evaluation of dialogues often shows insufficient correlation with human judgements due to the diversity of responses. |
| Approach: | They propose a framework that allows automatic evaluation of dialogues with human judgements. |
| Outcome: | The proposed framework achieves 99% evaluation accuracy with half the human effort spared. |
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. |
| Approach: | They propose to collect annotated human evaluation guidelines and a method for detecting guideline vulnerabilities using Large Language Models. |
| Outcome: | The proposed dataset includes eight vulnerabilities and a method for detecting guideline vulnerabilities. |
The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system, but its quality is often brought into question. |
| Approach: | They argue that the generation of more esoteric forms of language constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance. |
| Outcome: | The proposed system generates coherent and well-formed text of a particular type, usually given an input such as a prompt, outline, topic, or data. |
Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Literary tropes are at the crux of human imagination and communication. |
| Approach: | They propose to automatically transform similes from reddit to their literal counterparts using common sense knowledge to generate simile models. |
| Outcome: | The proposed method generates 88% novel similes that do not share properties with training data. |
LLM-REDIAL: A Large-Scale Dataset for Conversational Recommender Systems Created from User Behaviors with LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing CRS datasets suffer from data inextensibility and semantic inconsistency . |
| Approach: | They introduce the LLM-REDIAL dataset to facilitate the research in CRS by leveraging large language models to generate high-quality dialogues. |
| Outcome: | The proposed dataset is the largest multi-domain CRS dataset which consists of 47.6k multi-turn dialogues with 482.6k utterances across 4 domains. |
Finding a Balanced Degree of Automation for Summary Evaluation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Automated summarization metrics are reliable but often poorly correlated with human judgment. |
| Approach: | They propose a semi-automatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. |
| Outcome: | The proposed metrics are semi-automatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. |
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for evaluating the factuality of long-form text assume that every claim is verifiable. |
| Approach: | They propose a metric to evaluate factuality in diverse long-form generation tasks . they use open-weight language models to extract verifiable and unverifiably content . |
| Outcome: | The proposed metric can be implemented with either closed or fine-tuned open-weight language models. |
STORYSUMM: Evaluating Faithfulness in Story Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating abstractive summarization are lacking in faithfulness evaluation. |
| Approach: | They propose a dataset that measures faithfulness of LLM summaries with localized errors and faithfulness labels for evaluation methods. |
| Outcome: | The proposed method does not achieve more than 70% accuracy on this task. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |
CM-Gen: A Neural Framework for Chinese Metaphor Generation with Explicit Context Modelling (2022.coling-1)
Copied to clipboard
| Challenge: | Nominal metaphors are commonly used in human language and have been shown to be effective in persuading, expressing emotion, and stimulating interest. |
| Approach: | They propose a multitask framework which optimizes three tasks: NM identification, NM component identification, and NM generation. |
| Outcome: | The proposed framework outperforms baselines on consistency and creativity on the NM generation task in Chinese. |
Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)
Copied to clipboard
Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi
| Challenge: | a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics . |
| Approach: | They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context. |
| Outcome: | The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements. |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
SummScreen: A Dataset for Abstractive Screenplay Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing summarization datasets are constructed from various domains, such as news, and we characterize them using two entity-centric metrics. |
| Approach: | They propose to use a summarization dataset to evaluate TV series transcripts and recaps . they propose to employ two entity-centric metrics to evaluate the dataset . |
| Outcome: | The proposed model outperforms the existing model and its oracle counterparts in character overlap and accuracy. |
Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths (2020.coling-main)
Copied to clipboard
| Challenge: | Abstractive summarization at controllable lengths is a challenging task in natural language processing . high variance in screen-sizes often require extensive human supervision to perform these modifications. |
| Approach: | They propose a supervised method to construct abstractive summaries of a text document at controllable lengths using an interpretable multi-headed attention mechanism. |
| Outcome: | The proposed method outperforms baselines on two low-resource datasets in English by 14.70%. |
Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Evaluator groups such as domain experts, university students, and crowdworkers have been used to assess and compare chat-oriented dialogue systems. |
| Approach: | They analyze the impact of evaluator groups on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluer groups. |
| Outcome: | The proposed evaluations show that the evaluator group impact is not seen for Pairwise, and that it is beneficial for certain metrics. |
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation (2026.acl-long)
Copied to clipboard
Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett, Lucy Lu Wang
| Challenge: | a large-scale analysis of human evaluation protocols for long-form generation tasks is lacking in current practice . current protocols lack proper standardization and operationalization, which can limit validity of evaluation . |
| Approach: | They conduct a large-scale analysis of human evaluation protocols for long-form generation tasks in *CL conference papers from 2023–2025. |
| Outcome: | The proposed evaluation protocols lack standardization and operationalization, the authors show . they also find that the evaluation protocols are inadequate for specific domains and tasks . |
Generating Classical Chinese Poems from Vernacular Chinese (D19-1)
Copied to clipboard
| Challenge: | Existing models for classical Chinese poetry generation only allow users to use keywords to interfere with the meaning of generated poems. |
| Approach: | They propose a model to generate classical Chinese poems from vernacular . their model uses unsupervised machine translation to generate Chinese poems . human evaluation shows it can generate high-quality poems comparable to amateur poems - authors . |
| Outcome: | The proposed model improves the perplexity and BLEU of the proposed model compared with typical models and human evaluation shows it generates high-quality poems comparable to amateur poems. |
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics fail to align well with human judgments when evaluating QG questions. |
| Approach: | They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions. |
| Outcome: | The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency . |
MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in mobile Graphical User Interface (GUI) agents highlight the growing need for comprehensive evaluation benchmarks. |
| Approach: | They propose an online benchmark with 1080 tasks from 80 Chinese apps that measures task execution, complex reasoning, noise robustness and auto-eval framework with a reset mechanism. |
| Outcome: | The proposed benchmark measures task execution, complex reasoning, and noise robustness of agents by including 5 subsets, which set multiple evaluation dimensions. |
FaMeSumm: Investigating and Improving Faithfulness of Medical Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization models produce unfaithful outputs for medical text summarizing . a framework to improve faithfulness is proposed to improve medical text summary accuracy . |
| Approach: | They propose a framework to improve faithfulness by fine-tuning pre-trained language models based on medical knowledge. |
| Outcome: | The proposed framework improves faithfulness on medical summarization tasks. |
TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Long-form table question answering often generates paragraph long and complex answers . a prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context. |
| Approach: | They propose a modular framework that decomposes the whole process into three sub-modules . framework produces a QA-based plan first, followed by generating an answer conditioned on this plan . human evaluation results indicate the framework improves strong baselines on accuracy and truthfulness . |
| Outcome: | The proposed framework improves accuracy and truthfulness on the FeTaQA and QTSumm datasets. |
Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing discourse formalisms require large taxonomies of discourse relations to be accurate. |
| Approach: | They propose a linguistic framework for discourse analysis using questions under discussion . they propose qUD parser that derives a dependency structure of questions over full documents . |
| Outcome: | The proposed model is trained on a large, crowdsourced question-answering dataset. |
Rˆ3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on sarcasm generation focuses on context incongruity, but new work addresses this problem . |
| Approach: | They propose an unsupervised approach for sarcasm generation based on a non-sarcastic input sentence. |
| Outcome: | The proposed method generates sarcasm better than humans 34% of the time and better than a reinforced hybrid baseline 90% of the times. |
PathQG: Neural Question Generation from Facts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research for question generation encodes text as a sequence of tokens without explicitly modeling fact information. |
| Approach: | They propose to incorporate facts in the input text for question generation in a comprehensive way. |
| Outcome: | The proposed model outperforms state-of-the-art models and human evaluation shows it generates relevant and informative questions. |
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (2023.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
| Challenge: | a dataset of 1.5 million conversations distilled from everyday spoken situations is limited in scale due to its associated costs. |
| Approach: | They propose to make SODA a publicly available, million-scale high-quality social dialogue dataset . they contextualize social commonsense knowledge from a knowledge graph to distill broad spectrum of social interactions . |
| Outcome: | The proposed dataset is the first publicly available, million-scale high-quality social dialogue dataset. |
All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproduction (2026.acl-long)
Copied to clipboard
Ziyou Jiang, Mingyang Li, Junjie Wang, Yuekai Huang, Jie Huang, Zhiyuan Chang, Zhaoyang Li, Qing Wang
| Challenge: | Existing methods for harmful meme detection only learn the combination of harmful elements and lack understanding of these implicit expressions. |
| Approach: | They propose a method that detects harmful memes by replicating the design concept of malicious users. |
| Outcome: | The proposed method achieves the highest accuracy with 81.1% and has slight accuracy decreases when generalized to type-shifting and temporal-evolving memes. |
Can Large Language Models Be an Alternative to Human Evaluations? (2023.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is indispensable for assessing the quality of texts generated by machine learning models or written by humans. |
| Approach: | They propose to use large language models to evaluate unseen texts using the same instructions and samples . they also use LLMs to generate responses to questions that are used to conduct human evaluation . |
| Outcome: | The proposed model can be used to evaluate texts in open-ended story generation and adversarial attacks. |
Logic: Long-form Outline Generation via Imitative and Critical Self-refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for long-form outline generation have low knowledge density and lack detail . retrieval-augmented approaches struggle to maintain logical coherence across retrieved information . |
| Approach: | They propose a system that mimics human writers' refinement process by mimicking outlines through imitation and critical self-refinement. |
| Outcome: | The proposed system improves on the FreshWiki and WikiOutline datasets and establishes a coherent planning framework and structured knowledge base. |
Enhancing Human Evaluation in Machine Translation with Comparative Judgement (2025.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. |
| Approach: | They evaluate three annotation setups to integrate comparative judgment into human annotation for machine translation. |
| Outcome: | The proposed approach improves inter-annotator agreement and stability of the annotations. |
Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a low-resource natural language generation task requires a large number of examples to generate outputs and outputs. |
| Approach: | They propose a teacher-student pipeline that synthesizes accurate input–output pairs without human labels or parallel data. |
| Outcome: | The proposed pipeline synthesizes accurate input–output pairs without human labels or parallel data. |
FFAEval: Evaluating Dialogue System via Free-For-All Ranking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for open-domain dialogue systems show poor correlation with human assessment. |
| Approach: | They propose a free-for-all human evaluation framework that shares dialogue history with annotators for multi-turn scoring. |
| Outcome: | The proposed framework achieves a strong correlation with human assessment on English and Chinese dialogue systems. |
Kumatigi: Quality-Driven Data Augmentation for Low-Resource Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation for extremely low-resource languages faces compounding challenges: limited parallel data, orthographic inconsistency, and inconsistent metadata for principled training. |
| Approach: | They propose a quality-annotated French-Bambara corpus combining systematic curation with data augmentation strategies tailored to Bambaran. |
| Outcome: | The proposed framework achieves up to +3–4 BLEU over strong baselines. |
CPR-RAG: Clinical Prior-Regularized Retrieval for Anatomy-Aware 3D CT Report Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to grounding radiology reports from 3D volumetric data are limited due to visual-semantic ambiguity and lack of "normal" context. |
| Approach: | They propose a model-agnostic retrieval-augmented generation framework that integrates clinical priors into the retrieval process. |
| Outcome: | The proposed model improves clinical efficacy across state-of-the-art models. |
Synth-SBDH: A Synthetic Dataset of Social and Behavioral Determinants of Health for Clinical Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing SBDH datasets lack detailed annotations and are limited in their availability and coverage. |
| Approach: | They propose a synthetic SBDH annotation dataset with detailed SBDH status, temporal information, and rationale across 15 categories. |
| Outcome: | The proposed dataset outperforms models with no Synth-SBDH training on three tasks using real-world clinical datasets from two distinct hospital settings. |
DDxTutor: Clinical Reasoning Tutoring System with Differential Diagnosis-Based Structured Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have enabled various medical educational applications, but they often provide direct answers that could reduce students’ cognitive engagement and lead to fragmented learning. |
| Approach: | They propose a framework that follows differential diagnosis principles to decompose clinical reasoning into teachable components. |
| Outcome: | The proposed framework decomposes clinical reasoning into teachable components and generates structured teaching references and conducts diagnostic tutoring dialogues. |
VISTA: Verification In Sequential Turn-based Assessment (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics evaluate isolated responses or treat unverifiable content as errors, limiting their use for multi-turn dialogue. |
| Approach: | They propose a framework for evaluating conversational factuality via claim-level verification and sequential consistency tracking. |
| Outcome: | The proposed framework improves hallucination detection over existing benchmarks and models. |
Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models (2026.findings-acl)
Copied to clipboard
Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, Xiaolong Li
| Challenge: | Large language models (LLMs) have evolved from statistical sequence predictors to sophisticated autonomous agents capable of reasoning, planning, and sustaining multi-turn conversa-tions. |
| Approach: | They propose a system that instantiates a "Sentient Agent" that simulates human-like emotional changes and inner thoughts to provide a more realistic evaluation of the model in multi-turn conversations. |
| Outcome: | The proposed framework measures the agent's higher-order social cognition in multi-turn conversations. |
TripTide: A Benchmark for Adaptive Travel Planning under Disruptions (2026.findings-acl)
Copied to clipboard
Priyanshu Karmakar, Soumyabrata Chaudhuri, Shubhojit Mallick, Manish Gupta, Abhik Jana, Shreya Ghosh
| Challenge: | Recent work has shown the promise of Large Language Models (LLMs) for personalized, constraint-aware travel itinerary generation, but real-world travel often involves disruptions such as transit cancellations, weather-related closures, or overbooked attractions. |
| Approach: | They propose a benchmark to evaluate the ability of Large Language Models (LLMs) to revise travel itineraries under realistic disruptions. |
| Outcome: | The proposed benchmark evaluates the ability of Large Language Models (LLMs) to revise travel itineraries under real-world disruption scenarios. |